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INTERVENTION REPORT 


Supporting Postsecondary Success i e $ EDUCATION SCIENCES 


Open Learning Initiative (OLI) 


Online instruction, including online instruction that is blended with traditional in-person instruction, offers students more 
flexibility in the timing of their coursework and may deepen students’ understanding of the material. By leveraging online 
courseware, colleges have the potential to expand college access, reduce costs, recruit a more diverse student population, and 
accelerate instruction under some conditions.’ 


Carnegie Mellon University’s (CMU's) Open Learning Initiative (OLI) provides online courses and learning materials to instructors 
and learners at low or no cost that can be presented in blended or purely online formats. OL/ courses feature learning activities 
with immediate feedback for students and a dashboard for instructors to monitor student progress. 


This What Works Clearinghouse (WWC) report, part of the WWC's Supporting Postsecondary Success topic area, explores the 
effects of OL! on course completion rates and academic achievement. The WWC identified 12 studies of OL/, three of which 
meet WWC standards.’ The evidence presented in this report includes studies of the impacts of OL! on community college 
students, including students who were White, Black, Hispanic, and Asian or Pacific Islander. 


What Happens When Students Participate in OL/?? 


The evidence indicates that implementing OLI: 
* may increase credit accumulation and persistence 


¢ has inconsistent effects on academic achievement 


Findings on OLI from three studies that meet WWC standards are shown in Table 1. For each student outcome reviewed by 
the WWC, an effectiveness rating, the improvement index, and the number of studies and students that contributed to the 
findings are presented. The improvement index is a measure of the intervention’s effect on an outcome. It can be interpreted 
as the expected change in percentile rank for an average comparison group student if that student had received the 
intervention. Findings on credit accumulation are based on one study with 605 students. Findings on academic achievement 
are based on three studies with 795 students. See Box 1 for a description of WWC effectiveness ratings. 


Table 1. Summary of findings on OL/ from studies that meet WWC standards 


Evidence meeting 
Study findings WWC standards (version 3.0) 
Improvement index Number of Number of 
Outcome Effectiveness rating (percentile points) studies students 
Credit accumulation and persistence Potentially positive effects +/ ] 605 
Academic achievement Mixed effects +8 3 795 


Table Note: The improvement index is a measure of the effect of the intervention. The improvement index can be interpreted as the expected change in percentile rank for an average comparison 
group student if that student had received the intervention. For example, an improvement index of +7 means that the expected percentile rank of the average comparison group student would 
increase by 7 points if they received OL The improvement index values are generated by averaging findings from the outcome analyses that meet WWC standards, as reported by Bowen et al. 
(2014); Lovett, Meyer, & Thille (2008); and Schunn & Patchan (2009). Outcomes include completing and passing a course (credit accumulation and persistence) and final exam and standardized 
test scores (academic achievement). The effects of OL/ are not known for other outcomes within the Supporting Postsecondary Success topic area, including college access and enrollment; college 
attendance; college degree attainment; and labor market outcomes. 
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BOX 1. HOW THE WWC REVIEWS AND DESCRIBES EVIDENCE 


The WWC evaluates evidence based on the quality and results of reviewed studies. The criteria that the WWC uses for evaluating 
evidence are defined in the Procedures and Standards Handbooks and the Review Protocols. The studies summarized in this report 
were reviewed under WWC Standards (version 3.0) and the Supporting Postsecondary Success topic area protocol (version 3.0). 


To determine the effectiveness rating, the WWC considers what methods each study used, the direction of the effects, and how many 
studies tested the intervention. The higher the effectiveness rating, the more certain the WWC is about the reported results and about 
what will happen if the same intervention is implemented again. The following key provides a link between effectiveness ratings and 
the statements used in this report: 


Effectiveness Rating Rating Interpretation Description of the Evidence 


Positive (or Negative) Effects The intervention is likely to change an Strong evidence of a positive effect, with 
outcome no overriding contrary evidence 


Potentially Positive (or Negative) Effects § The intervention may change an outcome Evidence of a positive effect with no 
overriding contrary evidence 


No Discernible Effects The intervention may result in little to no No affirmative evidence of effects 
change in an outcome 


Mixed Effects The intervention has inconsistent effects Evidence includes studies in at least two 
on an outcome of these categories: studies with positive 
effects, studies with negative effects, or 
more studies with indeterminate effects 
than with positive or negative effects 


How is OL/ Implemented? 


The following section provides details of how OL! was implemented. This information can help educators identify the 
requirements for implementing OL/ and determine whether those implementation requirements would be feasible at their 
institutions. Information on OLI presented in this section comes from the three studies that meet WWC evidence standards 
(Bowen et al. 2014; Lovett, Meyer, & Thille, 2008; Schunn & Patchan, 2009), from OLI's website, and from correspondence with 
the developer. 


* Goal: OL! is designed to support learning and instruction via the high-quality online courses and learning materials available 
at low or no cost available on the OL/ website. 


¢ Target Population: Most OL/ courses are open to both students who take credit-bearing, instructor-led classes and 
independent learners who do not receive credit or instructor monitoring. Over the past decade, more than four million 
students have enrolled in 40 OL/ courses offered in both high schools and colleges.* 


« Method of Delivery: Courses and learning materials are available on the OL! website. While OL! content is delivered online, 
instructors may choose to supplement the online material with face-to-face reinforcement (hybrid delivery). Two studies 
featured in this report (Bowen et al., 2014; Lovett, Meyer, & Thille, 2008) investigated the effects of OL/ that used a blended 
approach, which supplemented online instruction provided through OLI/ with in-person instruction. One study (Schunn & 
Patchan, 2009) investigated the effects of OL/ using online instruction only. 


« Frequency and Duration of Service: OL! provides content that ranges in length from Comparison Group: In the 
several-hour modules to full-semester courses. Independent learners may complete three studies that contribute 
the material at their own pace, while students in instructor-led courses may be to this Intervention Report, 
assigned to complete the material in a specified timeframe. students in the comparison 


group completed 
coursework in traditional 
face-to-face settings. 


* Intervention Components: The OL/ model includes several components, which are 
noted in Table 2. 
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Table 2. Components of OL! 


Key component OLI 

Learning objectives Every OL! course, section, and page lists learning objectives, allowing students to assess whether they have understood key 
concepts. 

Expository materials Materials include readings, examples, and multi-media components that support learning objectives. These expository 
materials can be downloaded from OL/ to create a textbook. 

Learning activities with Two types of embedded learning activities provide opportunities for students to practice what they have learned and test their 

immediate feedback understanding: 


+ Learn by Doing activities include multiple choice questions, simulations, matching, and other options to help students 
practice a new skill or understand a new concept. 

+ Did! Get This? activities are self-assessments that pose questions to students to see if they understood a specific concept. 
These activities are presented after Learn by Doing activities. 

Students receive immediate feedback after each response addressing the misconception behind an incorrect answer or 

reinforcing the reasoning behind a correct answer. Checkpoint quizzes are also available within OL/, which provide detailed 

scores and information for instructors, and can be used to calculate course grades. 


Learning Dashboard The Learning Dashboard provides instructors with real-time student-level information about progress through OL course 
activities, interaction patterns with course materials, and class-level accuracy in answering questions related to each learning 
objective. 

Gradebook Student performance on quizzes is reported in the leaming dashboard as well as the gradebook. In the gradebook, instructors can 


view grades, adjust grades, change grading options, and grant exceptions to individual students. 


Face-to-face reinforcement — Some instructors supplement the online OL/ material with one to two hours of face-to-face instruction every week. 
(optional) 


What Does OLI Cost? 


The WWC identified a number of cost components from the OL! website. The cost breakdown below is not designed to be 
exhaustive; rather, it is designed to provide educators an overview of the kinds of resources needed to implement OLI. 


Personnel Costs: OL/ can be delivered as part of an academic course at a college. OL/ does not incur any personnel costs 
beyond those normally associated with teaching a course. 


Facilities Costs: OL/ courses delivered in a hybrid format require physical space to accommodate periodic face-to-face 
instruction. 


Equipment and Materials Costs: All OL! courses require internet access and an up-to-date web browser. Some content 
cannot be accessed on mobile devices. 


Costs Paid by Students or Parents: OL/ courses may be completed by independent learners or by students in 
credit-bearing, instructor-led classes. Most of the OL/ courses are free to independent learners, but a few charge a $10 
maintenance fee. Most OL/ courses delivered for credit will charge students a maintenance fee of $25, with fees ranging 
from $10 to $80 per student. Course fees may be paid by individual students or by colleges. Students are also subject to any 
tuition fees assessed by their college. 


In-Kind Supports: OL/ does not include any in-kind supports. 


Sources of Funding: The William and Flora Hewlett Foundation funded the development of the first four OL! courses 

at Carnegie Mellon University in 2002. Further development of OL/ courses was supported by the Bill & Melinda Gates 
Foundation, the Kresge Foundation, the Lumina Foundation, the National Science Foundation, the Spencer Foundation, and 
the Walter S. Johnson Foundation. Ongoing funding and operational support are provided by Carnegie Mellon University. 
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For More Information: 


About OL/ 
The Simon Initiative, Carnegie Mellon University 
5000 Forbes Avenue, Pittsburgh, PA 15213 
Web: http://oli.cmu.edu/. Phone: (412) 268-3294 


About the cost of the intervention 
For detailed cost information, please see: http://oli.cmu.edu/olis-payment-options-and-cost/. 


Research Summary 


The WWC identified 12 studies that investigated the effectiveness of OLI (Figure 1): 
+ 2studies meet WWC group design standards without reservations 

+ 1 study meets WWC group design standards with reservations 

+ 3studies do not meet WWC group design standards 

+ 6 Studies are ineligible for review 


The WWC reviews findings on an intervention’s effects on eligible outcome domains from studies that meet WWC group 
design standards, either with or without reservations. Based on this review, the WWC generates an effectiveness rating which 
summarizes how the intervention impacts, or changes, a particular outcome domain. Findings from studies that either do not 
meet WWC standards or are ineligible for review do not contribute to the effectiveness ratings. 


The three studies of OL/ that meet WWC group design standards reported findings on credit accumulation and persistence 
and academic achievement. The studies did not report findings on the following four outcome domains covered by the 
Supporting Postsecondary Success topic area: access and enrollment, college attendance, college degree attainment, and 
labor market outcomes. Citations for all 12 studies reviewed for this report are listed in the References section, which begins on 
page 10. 


Figure 1. Effectiveness ratings for OL! 


2 studies meet WWC study meets WWC i Ufol(=\-me (om ately studies are 
standards without standards with meet WWC Mate\ice ple) k-miels 
reservations reservations standards review 
Contributes to Effectiveness Ratings Do Not Contribute to Effectiveness Ratings 


— ee _______ 


OL! has potentially positive effects on credit accumulation and persistence 


The WWC determined that one study that meets WWC group design standards without reservations showed evidence of a positive and 
statistically significant effect of OL! on credit accumulation and persistence (Bowen et al., 2014). 


OL! has mixed effects on academic achievement 


The WWC determined that one study that meets WWC group design standards without reservations showed evidence of a positive effect 
of OLI on academic achievement (Lovett, Meyer, & Thille, 2008). The WWC determined that one study that meets WWC group design 
standards without reservations (Bowen et al., 2014) and one study that meets WWC group design standards with reservations 

(Schunn & Patchan, 2009) showed evidence of an indeterminate effect of OL! on academic achievement. 
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Main Finding 


s 


Table 3 shows the findings from three OL/ studies that meet WWC evidence standards, and includes WWC calculations of 

the mean difference, effect size, and the improvement index. Based on those studies, the effectiveness rating for the credit 
accumulation and persistence outcome is potentially positive effects, indicating that there is evidence of a positive effect with 
no overriding contrary evidence. This finding is based on 605 students. The effectiveness rating for the academic achievement 
outcomes is mixed effects, indicating evidence of inconsistent effects in the three studies reviewed. This finding is based on 795 


students. 


Table 3. Findings from studies of OL/ by outcome domain 


Mean 
(standard deviation) WWC calculations 

Intervention Comparison Mean Effect ~~ Improvement 
Measure (study) Study sample Sample size group group _ difference size index p-value 
Course completion rate (%) Postsecondary 605 students 87 82 5.0 0.23 +9 .004 
(Bowen et al., 2014)? students 
Course pass rate (6) Postsecondary 605 students 80 76 40 0.14 +6 .083 
(Bowen et al., 2014)? students 
Outcome average for credit accumulation and persistence across all studies 0.19 +7 
Comprehensive Assessment of Postsecondary 458 students 0.48 0.47 0.01 0.08 +3 420 
Outcomes in Statistics (CAOS) — students (0.15) (0.11) 
(Bowen et al., 2014)? 
Final exam score Postsecondary 431 students 0.57 0.55 0.02 0.11 +4 248 
(Bowen et al., 2014)? students (0.13) (0.22) 
Comprehensive Assessment of Postsecondary 61 students 68.00 53.00 15.00 0.55 421 044 
Outcomes in Statistics (CAOS) — students (18.90) (30.00) 
(Lovett, Meyer, & Thille, 
2008)° 
Final exam score Postsecondary 276 students 62.35 64.13 -1.78 — -0.07 -3 547] 
(Schunn & Patchan, 2009)° students (22.71) (25.68) 
Outcome average for academic achievement across all studies 0.19 +8 
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In What Context Was OL! Studied? 


The following section provides information on the setting and participants involved in the three studies of OL/ that meet WWC 
evidence standards. This information can help educators understand the context in which the studies of OL! were conducted 
so that they can better determine whether the program might be suitable for their setting. 
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3.0, page 26. 


WHERE THE STUDY WAS CONDUCTED 


3 studies, 942 students in 8 i Ss, es anc universities in Maryland, New York, and Pennsylvania 


< » 1 private 4-year college, 7 public 4-year colleges 


10 11 12 @ 


Postsecondary (PS) 


Details of Each Study that Met WWC Standards 


This section presents details for each study of OL/ that meets WWC standards. These details include the full study reference, 
findings summary, and description of study characteristics. A summary of findings for each outcome domain examined is 
presented in the first table for each study. The second table for each study provides a description of the study characteristics. 
These study-level details include contextual information on the study setting, methods, sample, intervention group, 
comparison group, outcomes, and implementation details. For additional information, the reader should refer to the original 
studies. 


Research details for Bowen et al. (2014) 


Bowen, W. G., Chingos, M. M., Lack, K. A., & Nygren, T. |. (2014). Interactive learning online at public universities: Evidence from a 
six-campus randomized trial. Journal of Policy Analysis and Management, 33(1), 94-111. Retrieved from 
https://eric.ed.gov/?id=EJ1027704 


Additional Source: 


Bowen, W. G., Chingos, M. M., Lack, K. A., & Nygren, T. |. (2012). Interactive learning online at public universities: Evidence 
from randomized trials. New York: Ithaka S+R. doi:10.18665/sr.22464 


Findings from Bowen et al. (2014) show evidence of a positive effect of OL! on credit accumulation and persistence (Table 4). 
The improvement index is a measure of the effect of the intervention. For example, an improvement index of +7 means that 
the percentile rank of the average comparison group student would improve by 7 points on credit accumulation outcomes 
if they received OLI. This finding is based on two outcomes and 605 students each. Findings on academic achievement 
outcomes show evidence of indeterminate effects. These findings are based on two outcomes, with 458 and 431 students, 


respectively. 
Table 4. Summary of findings from Bowen et al. (2014) Meets WWC group design standards with reservations 
Study findings 
Outcome domain Sample size Average effect size Improvement index Statistically significant 
Credit accumulation and persistence 7 colleges and universities/ +0.19 +/ Yes 
605 students 
Academic achievement 7 colleges and universities/ +0.09 +4 No 
458 students 
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Table 5. Description of study characteristics for Bowen et al. (2014) 


WWC evidence 
rating 


Setting 


Methods 


Study sample 


Intervention 
group 


Comparison 
group 


Outcomes and 
measurement 


Additional 


implementation 


details 


Meets WWC Group Design Standards Without Reservations. [his is a randomized controlled trial (RCT) with low 
attrition. 


The study took place in seven public, 4-year colleges. Four colleges were located in New York (University of Albany, SUNY Institute 
of Technology, Baruch College, and City College), and three colleges were located in Maryland (University of Maryland, University of 
Maryland Baltimore County, and Towson University). 


The study uses a randomized controlled trial (RCT) design. Students who were registered for introductory statistics courses in the 
seven participating colleges were asked to participate in the study at the beginning of the semester. Those that agreed were given 

a modest incentive, filled out a baseline survey, and were randomly assigned to either a hybrid classroom format with OL/or the 
traditional classroom format. Students completed the Comprehensive Assessment of Outcomes in Statistics (CAOS) at the beginning 
and end of the semester. 


The sample included 605 students who were randomly assigned, with 313 students in the intervention group and 292 students in 
the comparison group. There was no attrition in the study. 


In the intervention group, the study participants were 39 percent male, 46 percent White, had an average age of 22.0 years, and 
50 percent had a family income of less than $50,000. In the comparison group, the study participants were 46 percent male, 41 
percent White, had an average age of 21.9 years, and 49 percent had a family income of less than $50,000. 


Students in the intervention group participated in Interactive Learning Online (ILO), which was a rebranded OL/intervention that 
provided the same instruction as OLI. ILO sections were delivered in a hybrid mode, in which most of the instruction was delivered 
through interactive online materials, but the online instruction was supplemented by a weekly one-hour face-to-face session, so 
that students could ask questions or be given targeted assistance by the instructor. The /LO course for this study included textual 
explanations of concepts, worked examples, and practice problems. Students were also required to manipulate data using statistical 
software packages. 


Students assigned to the comparison group were taught the introductory statistics course as it is usually offered at their college, 
with face-to-face instruction. 


The study measures whether or not students completed the introductory statistics course and whether they passed the course. A 
student can complete a course without passing it if he/she is enrolled until the end of the semester but received a failing grade. 
These two outcomes fall under the credit accumulation and persistence domain. The study also measured academic achievement at 
the end of the semester using the Comprehensive Assessment of Outcomes in Statistics (CAOS), which is a 40-item multiple-choice, 
standardized measure of statistical literacy and reasoning skills.° The authors also report findings on the final exam from the course 
as well. Both of these outcomes fall under the academic achievement domain. 


The study does not provide details on implementation support for LO. 


Research details for Lovett, Meyer, & Thille (2008) 


Lovett, M., Meyer, O., & Thille, C. (2008). The Open Learning Initiative: Measuring the effectiveness of the OL/ statistics course in 
accelerating student learning. Journal of Interactive Media in Education, 1. Retrieved from 
https://eric.ed.gov/?id=EJ840810 


Findings from Lovett, Meyer, & Thille (2008) show evidence of a positive effect of OL! on academic achievement (Table 6). The 
improvement index is a measure of the effect of the intervention. For example, an improvement index of +21 means that the 
percentile rank of the average comparison group student would improve by 21 points on academic achievement outcomes if 
they received OL/. This finding is based on one outcome and 61 students. 
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Table 6. Summary of findings from Lovett, Meyer, & Thille (2008) | Meets WWC group design standards with reservations 


Study findings 
Outcome domain Sample size Average effect size Improvement index Statistically significant 
Academic achievement | university/61 students +0.55 +21 Yes 
Table 7. Description of study characteristics for Lovett, Meyer, & Thille (2008) 
WWC evidence Meets WWC Group Design Standards Without Reservations. [his is a randomized controlled trial (RCT) with low 
rating attrition. The authors also describe two quasi-experimental design (QED) studies that do not meet WWC Group Design Standards 


because they do not establish baseline equivalence of intervention and comparison groups. 
Setting The study took place in spring 2007 in an introductory statistics course at Carnegie Mellon University. 


Methods A month before the semester began, students were invited to participate in OL Of the 68 students who volunteered, 22 students 
were randomly selected for the intervention. The remaining 46 students formed the comparison group, with 4 students dropping 
out before the course began. 


Study sample The analytic sample included 61 students, with 21 students in the intervention group and 40 students in the comparison group. The 
study does not provide demographic or other characteristics of the students who participated in the study. 


Intervention Students in the intervention group enrolled in a college-level, non-calculus-based introduction to statistics course. The OL/ 
group intervention condition involved (a) working in an online learning environment (OL) to acquire most of the course content, (b) 
meeting with an instructor approximately two times a week for 50-minute sessions to ask questions and review more challenging 


material, and (c) completing the semester's material in approximately half the regularly scheduled time (8 weeks instead of 15 
weeks). 
Comparison Students in the comparison group attended a traditional, classroom-based introductory statistics course. The course’s content was 
group the same as that delivered in the intervention condition. 
Outcomes and The Comprehensive Assessment of Outcomes in Statistics (CAOS) is a 40-item multiple-choice, standardized measure of statistical 
measurement literacy and reasoning skills.° This outcome falls under the academic achievement domain. It was measured at the beginning and 


end of the semester. The authors also provided final exam scores; however, since different versions of the exam were administered to 
the intervention and comparison group, this outcome is ineligible for review. 


Additional The study does not provide details on implementation support for OLI. 
implementation 
details 


Research details for Schunn & Patchan (2009) 


Schunn, C. D., & Patchan, M. (2009). An evaluation of accelerated learning in the CMU Open Learning Initiative course “Logic & 
Proofs” Pittsburgh, PA: University of Pittsburgh, Learning Research and Development Center. 


Findings from Schunn & Patchan (2009) show evidence of an indeterminate effect of OL! on academic achievement (Table 8). 
This finding is based on one outcome and 276 students. 


Table 8. Summary of findings from Schunn & Patchan (2009) Meets WWC group design standards with reservations 
Study findings 
Outcome domain Sample size Average effect size Improvement index Statistically significant 
Academic achievement | university/276 students -0.07 3 No 
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Table 9. Description of study characteristics for Schunn & Patchan (2009) 


WWC evidence Meets WWC Group Design Standards With Reservations. This is a quasi-experimental design (QED), with baseline 

rating equivalence established between the intervention and comparison groups. 

Setting The study took place at Carnegie Mellon University in four sections of a symbolic logic course during the fall 2007 and spring 2008 
semesters. 

Methods The study is a quasi-experimental design (QED). Students opted to either participate in an online (OL) section or in a traditional 
face-to-face section of a symbolic logic course. The same instructor taught both sections of the course each semester. 

Study sample The analytic sample for the fall 2007 semester included 88 students, with 45 students in the intervention group and 43 students 
in the comparison group. The analytic sample for the spring 2008 semester included 188 students, with 83 students in the 
intervention group and 105 students in the comparison group. 
Detailed demographic breakdowns are not reported, but the authors did indicate that there were no significant differences between 
the intervention and comparison group on gender, year, GPA, or prior experience with logic courses and online instruction in 
general. 

Intervention Students in the intervention group received online instruction through OL/ that generally mirrored the content of the course taught 

group in the traditional (comparison) sections. The authors note that the OL/sections included several additional topics not covered in the 
comparison sections. 

Comparison Students in the comparison group received instruction in the traditional face-to-face format. 

group 

Outcomes and The authors report findings on the final exam from the course. This outcome falls under the academic achievement domain. 

measurement Baseline equivalence was established on cumulative GPA prior to the semester when the study took place. 

Additional The study does not provide details on implementation support for OL/. 

implementation 

details 
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4 Bier, N., Moore, S., & Van Velsen, M. (2019). Instrumenting courseware and leveraging data with the Open Learning Initiative (OLI). Companion Proceed- 
ings 9th International Learning Analytics & Knowledge Conference, Tempe, AZ. 


wu 


Delmas, G., Joan, G., Ooms, A., & Chance, B. (2007). Assessing Students’ Conceptual Understanding after a First Course in Statistics. Statistics Education 
Research Journal, 6(2), 28-58. 


a 


Delmas et al. (2007). 


Recommended Citation 


What Works Clearinghouse, Institute of Education Sciences, U.S. Department of Education. (2020, January). Supporting Postsecondary Success intervention 
report: Open Learning Initiative (OLI). Retrieved from https://whatworks.ed.gov 


OPEN LEARNING INITIATIVE | JANUARY 2020 11 


